Assumption-checking rather than (just) testing: The importance of visualization and effect size in statistical diagnostics.

Statistical methods generally have assumptions (e.g., normality in linear regression models). Violations of these assumptions can cause various issues, like statistical errors and biased estimates, whose impact can range from inconsequential to critical. Accordingly, it is important to check these assumptions, but this is often done in a flawed way. Here, I first present a prevalent but problematic approach to diagnostics-testing assumptions using null hypothesis significance tests (e.g., the Shapiro-Wilk test of normality). Then, I consolidate and illustrate the issues with this approach, primarily using simulations. These issues include statistical errors (i.e., false positives, especially with large samples, and false negatives, especially with small samples), false binarity, limited descriptiveness, misinterpretation (e.g., of p-value as an effect size), and potential testing failure due to unmet test assumptions. Finally, I synthesize the implications of these issues for statistical diagnostics, and provide practical recommendations for improving such diagnostics. Key recommendations include maintaining awareness of the issues with assumption tests (while recognizing they can be useful), using appropriate combinations of diagnostic methods (including visualization and effect sizes) while recognizing their limitations, and distinguishing between testing and checking assumptions. Additional recommendations include judging assumption violations as a complex spectrum (rather than a simplistic binary), using programmatic tools that increase replicability and decrease researcher degrees of freedom, and sharing the material and rationale involved in the diagnostics.

Visualizing Pollution: Representations of Biological Data in Water Pollution Control in the United States, 1948-1962.

After the United States Congress passed the Water Pollution Control Act of 1948, biologists played an increasingly significant role in scientific studies of water pollution. Biologists interacted with other experts, notably engineers, who managed the public agencies devoted to water pollution control. Although biologists were at first marginalized within these agencies, the situation began to change by the early 1960s. Biological data became an integral part of water pollution control. While changing societal values, stimulated by an emerging ecological awareness, may explain broader shifts in expert opinion during the 1960s, this article explores how graphs changed experts' perceptions of water pollution. Experts communicated with each other via reports, journal articles, and conference speeches. Those sources reveal that biologists began experimenting with new graphical methods to simplify the complex ecological data they collected from the field. Biologists, I argue, followed the engineers' lead by developing graphical methods that were concise and quantitative. Their need to collaborate with engineers forced them to communicate, negotiate, and overcome conflicts and misunderstandings. By meeting engineers' expectations and promoting the value of their data through images as much as words, biologists asserted their authority within water pollution control by the early 1960s.

Heterogeneity in meta-analysis: a comprehensive overview.

In recent years, meta-analysis has evolved to a critically important field of Statistics, and has significant applications in Medicine and Health Sciences. In this work we briefly present existing methodologies to conduct meta-analysis along with any discussion and recent developments accompanying them. Undoubtedly, studies brought together in a systematic review will differ in one way or another. This yields a considerable amount of variability, any kind of which may be termed heterogeneity. To this end, reports of meta-analyses commonly present a statistical test of heterogeneity when attempting to establish whether the included studies are indeed similar in terms of the reported output or not. We intend to provide an overview of the topic, discuss the potential sources of heterogeneity commonly met in the literature and provide useful guidelines on how to address this issue and to detect heterogeneity. Moreover, we review the recent developments in the Bayesian approach along with the various graphical tools and statistical software that are currently available to the analyst. In addition, we discuss sensitivity analysis issues and other approaches of understanding the causes of heterogeneity. Finally, we explore heterogeneity in meta-analysis for time to event data in a nutshell, pointing out its unique characteristics.

Conceptual framework for neuronal ensemble identification and manipulation related to behavior using calcium imaging.

Significance: The identification and manipulation of spatially identified neuronal ensembles with optical methods have been recently used to prove the causal link between neuronal ensemble activity and learned behaviors. However, the standardization of a conceptual framework to identify and manipulate neuronal ensembles from calcium imaging recordings is still lacking. Aim: We propose a conceptual framework for the identification and manipulation of neuronal ensembles using simultaneous calcium imaging and two-photon optogenetics in behaving mice. Approach: We review the computational approaches that have been used to identify and manipulate neuronal ensembles with single cell resolution during behavior in different brain regions using all-optical methods. Results: We proposed three steps as a conceptual framework that could be applied to calcium imaging recordings to identify and manipulate neuronal ensembles in behaving mice: (1) transformation of calcium transients into binary arrays; (2) identification of neuronal ensembles as similar population vectors; and (3) targeting of neuronal ensemble members that significantly impact behavioral performance. Conclusions: The use of simultaneous two-photon calcium imaging and two-photon optogenetics allowed for the experimental demonstration of the causal relation of population activity and learned behaviors. The standardization of analytical tools to identify and manipulate neuronal ensembles could accelerate interventional experiments aiming to reprogram the brain in normal and pathological conditions.

Computational Methodology for Peptide Vaccine Design for Zika Virus: A Bioinformatics Approach.

With the increasing frequency of viral epidemics, vaccines to augment the human immune response system have been the medium of choice to combat viral infections. The tragic consequences of the Zika virus pandemic in South and Central America a few years ago brought the issues into sharper focus. While traditional vaccine development is time-consuming and expensive, recent advances in information technology, immunoinformatics, genetics, bioinformatics, and related sciences have opened the doors to new paradigms in vaccine design and applications.Peptide vaccines are one group of the new approaches to vaccine formulation. In this chapter, we discuss the various issues involved in the design of peptide vaccines and their advantages and shortcomings, with special reference to the Zika virus for which no drugs or vaccines are as yet available. In the process, we outline our work in this field giving a detailed step-by-step description of the protocol we follow for such vaccine design so that interested researchers can easily follow them and do their own designing. Several flowcharts and figures are included to provide a background of the software to be used and results to be anticipated.

A new test and graphical tool to assess the goodness of fit of logistic regression models.

A prognostic model is well calibrated when it accurately predicts event rates. This is first determined by testing for goodness of fit with the development dataset. All existing tests and graphic tools designed for the purpose suffer several drawbacks, related mainly to the subgrouping of observations or to heavy dependence on arbitrary parameters. We propose a statistical test and a graphical method to assess the goodness of fit of logistic regression models, obtained through an extension of similar techniques developed for external validation. We analytically computed and numerically verified the distribution of the underlying statistic. Simulations on a set of realistic scenarios show that this test and the well-known Hosmer-Lemeshow approach have similar type I error rates. The main advantage of this new approach is that the relationship between model predictions and outcome rates across the range of probabilities can be represented in the calibration belt plot, together with its statistical confidence. By readily spotting any deviations from the perfect fit, this new graphical tool is designed to identify, during the process of model development, poorly modeled variables that call for further investigation. This is illustrated through an example based on real data.

Maximizing the Yield of Small Samples in Prevention Research: A Review of General Strategies and Best Practices.

The goal of this manuscript is to describe strategies for maximizing the yield of data from small samples in prevention research. We begin by discussing what "small" means as a description of sample size in prevention research. We then present a series of practical strategies for getting the most out of data when sample size is small and constrained. Our focus is the prototypic between-group test for intervention effects; however, we touch on the circumstance in which intervention effects are qualified by one or more moderators. We conclude by highlighting the potential usefulness of graphical methods when sample size is too small for inferential statistical methods.

Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers.

Predicting the probability of the occurrence of a binary outcome or condition is important in biomedical research. While assessing discrimination is an essential issue in developing and validating binary prediction models, less attention has been paid to methods for assessing model calibration. Calibration refers to the degree of agreement between observed and predicted probabilities and is often assessed by testing for lack-of-fit. The objective of our study was to examine the ability of graphical methods to assess the calibration of logistic regression models. We examined lack of internal calibration, which was related to misspecification of the logistic regression model, and external calibration, which was related to an overfit model or to shrinkage of the linear predictor. We conducted an extensive set of Monte Carlo simulations with a locally weighted least squares regression smoother (i.e., the loess algorithm) to examine the ability of graphical methods to assess model calibration. We found that loess-based methods were able to provide evidence of moderate departures from linearity and indicate omission of a moderately strong interaction. Misspecification of the link function was harder to detect. Visual patterns were clearer with higher sample sizes, higher incidence of the outcome, or higher discrimination. Loess-based methods were also able to identify the lack of calibration in external validation samples when an overfit regression model had been used. In conclusion, loess-based smoothing methods are adequate tools to graphically assess calibration and merit wider application.

Advanced multiplicity adjustment methods in clinical trials.

During the last decade, many novel approaches for addressing multiplicity problems arising in clinical trials have been introduced in the literature. These approaches provide great flexibility in addressing given clinical trial objectives and yet maintain strong control of the familywise error rate. In this tutorial article, we review multiple testing strategies that are related to the following: (a) recycling local significance levels to test hierarchically ordered hypotheses; (b) adapting the significance level for testing a hypothesis to the findings of testing previous hypotheses within a given test sequence, also in view of certain consistency requirements; (c) grouping hypotheses into hierarchical families of hypotheses along with recycling the significance level between those families; and (d) graphical methods that permit repeated recycling of the significance level. These four different methodologies are related to each other, and we point out some connections as we describe and illustrate them. By contrasting the main features of these approaches, our objective is to help practicing statisticians to select an appropriate method for their applications. In this regard, we discuss how to apply some of these strategies to clinical trial settings and provide algorithms to calculate critical values and adjusted p-values for their use in practice. The methods are illustrated with several numerical examples.

Graphical tools for model selection in generalized linear models.

Model selection techniques have existed for many years; however, to date, simple, clear and effective methods of visualising the model building process are sparse. This article describes graphical methods that assist in the selection of models and comparison of many different selection criteria. Specifically, we describe for logistic regression, how to visualize measures of description loss and of model complexity to facilitate the model selection dilemma. We advocate the use of the bootstrap to assess the stability of selected models and to enhance our graphical tools. We demonstrate which variables are important using variable inclusion plots and show that these can be invaluable plots for the model building process. We show with two case studies how these proposed tools are useful to learn more about important variables in the data and how these tools can assist the understanding of the model building process.

SU-E-T-259: A Statistical and Machine Learning-Based Tool for Modeling and Visualization of Radiotherapy Treatment Outcomes.

PURPOSE: Effective radiotherapy outcomes modeling could provide physicians with better understanding of the underlying disease mechanism, enabling to early predict outcomes and ultimately allowing for individualizing treatment for patients at high risk. This requires not only sophisticated statistical methods, but user-friendly visualization and data analysis tools. Unfortunately, few tools are available to support these requirements in radiotherapy community. METHODS: Our group has developed Matlab-based in-house software called DREES for statistical modeling of radiotherapy treatment outcomes. We have noticed that advanced machine learning techniques can be used as useful tools for analyzing and modeling the outcomes data. To this end, we have upgraded DREES such that it takes advantage of useful Statistics and Bioinformatics toolboxes in Matlab that provide robust statistical data modeling and analysis methods as well as user-friendly visualization and graphical interface. RESULTS: Newly added key features include variable selection, discriminant analysis and decision tree for classification, and k-means and hierarchical clustering functions. Also, existing graphical tools and statistical methods in DREES were replaced with a library of the Matlab toolboxes. We analyzed several radiotherapy outcomes datasets with our tools and showed that these can be effectively used for building normal tissue complication probability (NTCP) and tumor control probability (TCP) models. CONCLUSIONS: We have developed an integrated software tool for modeling and visualization of radiotherapy outcomes data within the Matlab programming environment. It is our expectation that this tool could help physicians and scientists better understand the complex mechanism of disease and identify clinical and biological factors related to outcomes.

SU-E-T-473: Performance Assessment of the TOPAS Tool for Particle Simulation for Proton Therapy Applications.

PURPOSE: The TOPAS Tool for Particle Simulation was developed to make Geant4 Monte Carlo simulation more readily available for research and clinical physicists. Before releasing this new tool to the proton therapy community, several test have been performed to ensure accurate simulations in a variety of proton therapy setups. METHODS: TOPAS can model a passive scattering or scanning beam treatment head, model a patient geometry based on CT images, score dose, fluence, etc., save and replay a phase space, provides advanced graphics, and is fully four-dimensional (4D) to handle variations in beam delivery and patient geometry during treatment. An innovative control system meets requirements for ease of use, reliability and repeatability without sacrificing flexibility. To test the TOPAS code, we modeled proton therapy treatment examples including the UCSF eye treatment beamline (UCSFETB), the MGH STAR radiosurgery beamline and the MGH gantry treatment head in passive scattering and scanning modes. The simulations included time-dependent geometry and time- dependent beam current delivery. RESULTS: At the UCSFETB, time- dependent depth dose distributions were accurately simulated with time- varying energy modulation from a rotating propeller. At the MGH STAR beamline, distal and proximal ranges agreed within measurement uncertainty and the shape of the simulated SOBP followed measured data. For the MGH gantry treatment head in passive scattering mode, SOBPs were simulated for the full set of range modulator wheel and second scatterer combinations. TOPAS simulation was within clinical required accuracy. For the MGH nozzle in scanning mode, a variety of scan patterns were simulated with fluence maps generated for cases including beam current modulation, energy modulation and target tracking. CONCLUSIONS: Our results demonstrate the functionality of TOPAS. They show agreement with measured data and demonstrate the capabilities of TOPAS in simulating beam delivery in 3D and 4D. This work was supported by IH/NCI under R01 CA 140735-01.

SU-E-I-28: Development of Graphic Patient Models for a Real-Time Skin Dose Tracking System (DTS) for Fluoroscopic Interventional Procedures.

PURPOSE: To develop a library of graphic human models that closely match patients undergoing interventional fluoroscopic procedures in order to obtain an accurate estimate of their skin dose. METHODS: A dose tracking system (DTS) has been developed that calculates the dose to the patient's skin in real time during fluoroscopic procedures based on a graphical simulation of the x-ray system and the patient. The calculation is performed using a lookup table containing values of mGy per mAs at a reference point and inverse-square correction using the distance from the source to individual points on the skin. For proper inverse-square correction, the external shape of the graphic should closely match that of the patient. We are in the process of developing a library of 3D human graphic models categorized as a function of basic body type, sex, height and weight. Two different open- source software applications are being used to develop graphic models with varying weights and heights, to 'morph' the shapes for body type and to 'pose' them for proper positioning on the table. The DTS software is being designed such that the most appropriate body graphic can be automatically selected based on input of several basic patient dimensional metrics. RESULTS: A series of male and female body graphic models have been developed which vary in weight and height. Matching pairs have been constructed with arms at the side and over the head to simulate the usual placement in cardiac procedures. The error in skin dose calculation due to inverse-square correction is expected to be below 5% if the graphic can match the position of the patient's skin surface within 1 cm. CONCLUSIONS: A library of categorized body shapes should allow close matching of the graphic to the patient shape allowing more accurate determination of skin dose with the DTS. Support for this work was provided in part by NIH grants R43FD0158401, R44FD0158402, R01EB002873 and R01EB008425, and by Toshiba Medical Systems Corporation.

An introduction to causal inference.

This paper summarizes recent advances in causal inference and underscores the paradigmatic shifts that must be undertaken in moving from traditional statistical analysis to causal analysis of multivariate data. Special emphasis is placed on the assumptions that underlie all causal inferences, the languages used in formulating those assumptions, the conditional nature of all causal and counterfactual claims, and the methods that have been developed for the assessment of such claims. These advances are illustrated using a general theory of causation based on the Structural Causal Model (SCM) described in Pearl (2000a), which subsumes and unifies other approaches to causation, and provides a coherent mathematical foundation for the analysis of causes and counterfactuals. In particular, the paper surveys the development of mathematical tools for inferring (from a combination of data and assumptions) answers to three types of causal queries: those about (1) the effects of potential interventions, (2) probabilities of counterfactuals, and (3) direct and indirect effects (also known as "mediation"). Finally, the paper defines the formal and conceptual relationships between the structural and potential-outcome frameworks and presents tools for a symbiotic analysis that uses the strong features of both. The tools are demonstrated in the analyses of mediation, causes of effects, and probabilities of causation.

Beanplot uma nova ferramenta gráfica / Beanplot a new graphical tool

A estatística descritiva é uma poderosa ferramenta para se analisar conjuntos de dados, entretanto é muito pouco utilizada. Uma análise descritiva bem conduzida pode evitar vários problemas que podem ocorrer em análises mais complexas, além de fornecer um retrato da amostra em estudo. Na estatística descritiva existem os métodos gráficos, que se bem empregados, são bem mais informativos que tabelas. Dentre os tipos de gráficos mais conhecidos existem o boxplot, histograma, gráfico de dispersão, ou de barras. O objetivo desse artigo é descrever um novo tipo de gráfico chamado beanplot que pode ser feito no aplicativo R. Através de exemplos é mostrado como fazer o beanplot no R e como interpretar seus resultados. Nesse gráfico podemos representar várias informações sobre variáveis quantitativas, tais como: média, mediana, distribuição dos dados, etc. Além disso, através desse gráfico podemos comparar distribuições de diversas variáveis ou da mesma variável em diferentes grupos.

Descriptive analysis is a powerful tool to analyze data sets, but is rarely used. It can avoid many problems that can occur in more complex analyses, providing a picture of the sample under study. Some graphical methods are much more informative than tables. There are several types of graphics which are well known: boxplot, histogram, scatter plot, or bar plot. The aim of this paper is to present a new type of graph called beanplot, describing the steps to build the graphs using the statistical software R. Besides, some examples are presented to discuss how to interpret the results. Through beanplot graphs, it is possible to represent a plenty of information regarding quantitative variables, such as mean, median, distribution of data, etc. Moreover, through this graphic we compare distributions of several variables or the same variable in different groups.